Repository navigation
Decouple gRPC execution from workflow contexts - #1842
geomagilles wants to merge 2 commits into
Conversation
ba13c03 to
4fbfa56
Compare
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are reported for the extraction of reusable gRPC execution into a standalone client.
No open issues 🎉
What changed in this PR
Extracts dynamic gRPC execution into a standalone client for use by other workflow engines, while retaining SDK-specific behavior in the existing adapter. This prepares Lemline integration without implementing it or changing HTTP support.
Changes:
- Adds reusable asynchronous gRPC calls and Protobuf JSON conversion without workflow-engine dependencies.
- Refactors the workflow adapter to delegate transport execution while preserving result shapes, channel providers and positioned errors.
- Adds transport lifecycle and conversion tests, module wiring and usage documentation.
| File | Description |
|---|---|
impl/test/src/test/java/org/openworkflow/sdk/impl/test/grpc/GrpcUnaryTest.java |
Tests channel-provider reuse and positioned transport errors. |
impl/README.md |
Links to standalone client documentation. |
impl/pom.xml |
Registers and manages the new client module. |
impl/grpc/src/main/java/org/openworkflow/sdk/impl/executors/grpc/ProtobufMessageUtils.java |
Delegates Protobuf JSON conversion to the shared utility. |
impl/grpc/src/main/java/org/openworkflow/sdk/impl/executors/grpc/ModelStreamObserver.java |
Removes the workflow-bound observer base. |
impl/grpc/src/main/java/org/openworkflow/sdk/impl/executors/grpc/ItemStreamObserver.java |
Removes the superseded unary result observer. |
impl/grpc/src/main/java/org/openworkflow/sdk/impl/executors/grpc/GrpcExecutor.java |
Delegates RPC execution and adapts workflow inputs, results and errors. |
impl/grpc/src/main/java/org/openworkflow/sdk/impl/executors/grpc/CollectionStreamObserver.java |
Removes the superseded streaming collection observer. |
impl/grpc/pom.xml |
Adds the standalone client dependency. |
impl/grpc-client/src/test/java/org/openworkflow/sdk/grpc/ProtobufJsonTest.java |
Tests JSON conventions and unknown-field rejection. |
impl/grpc-client/src/test/java/org/openworkflow/sdk/grpc/DynamicGrpcCallTest.java |
Tests RPC modes, options, errors, cancellation and channel ownership. |
impl/grpc-client/src/main/java/org/openworkflow/sdk/grpc/ProtobufJson.java |
Adds workflow-independent Protobuf JSON conversion. |
impl/grpc-client/src/main/java/org/openworkflow/sdk/grpc/DynamicGrpcCall.java |
Implements reusable dynamic RPCs with buffered responses. |
impl/grpc-client/README.md |
Documents usage, ownership, options and streaming limits. |
impl/grpc-client/pom.xml |
Defines client dependencies and aligns the Protobuf runtime. |
Review effort: Balanced
Rebuild the standalone gRPC extraction on current main under common/grpc and publish openworkflow-common-grpc through the common parent and root dependency management. Retain the existing Gson and Protobuf runtime alignment when leaving the implementation parent. Expose WorkflowGrpcCall and WorkflowProtobufJson using native grpc-java and Protobuf values. Keep workflow resources, expressions, model conversion, positioned errors and channel-provider policy in the existing GrpcExecutor adapter. Preserve its public constructor and result shapes, including the existing client/bidi argument convention. Add no request/response DTO hierarchy or payload cloning. Coordinate terminal delivery before cancellation callbacks, stop unfinished RPCs on local future timeouts or asynchronous completion, and release owned channels. Preserve supplier failure semantics and release raw transport references after SDK adaptation. Cover synchronous handlers, competing completions, native cancellation, deadlines, borrowed-channel reuse, status/cause/trailers and owned transport termination. Document reusable execution components as the third SDK usage in the root README, update the compiled common gRPC example and use generic engine descriptions. Keep per-message conversion, backpressure and stream resumption for separate work. The minimal common parent can be reconciled with the HTTP/OpenAPI reorganization when those changes merge. Validate 44 focused gRPC cases and the full reactor: 41 successful modules, 815 passing cases and 17 existing skips. Check repository formatting/style, common Javadocs, native dependency boundaries, BOM publication and the standalone README example. Retain validation and review logs, including intermediate timing-sensitive failures in unchanged tests. Signed-off-by: Gilles Barbier <gilles@infinitic.io>
4fbfa56 to
163750c
Compare
Merge the upstream SDK package rename into the gRPC extraction. Update the standalone client, workflow adapter, tests, documentation, and Maven coordinates to use cloud.openworkflow.sdk.
|
@fjtirado I’ve updated this gRPC PR to follow the same approach we discussed for HTTP and OpenAPI in #1844. The reusable gRPC client now lives in common/grpc, so other workflow engines can use it directly. The reference engine’s workflow-specific logic stays in impl/grpc, which depends on the shared client. I’ve also applied the namespace change from #1858. The Maven coordinate for the shared client is now cloud.openworkflow.sdk:openworkflow-common-grpc. The full Maven build passed locally on JDK 21, including all 44 focused gRPC tests. Thanks for the feedback! |
Why
Other workflow engines need to reuse dynamic gRPC execution without constructing SDK workflow applications, definitions, contexts or models. This extracts a reusable component while keeping the reference engine's workflow policy in its adapter.
Approach
The gRPC extraction follows the shared-components direction discussed in #1844 and incorporates the SDK namespace migration from #1858.
common/grpc, with artifactcloud.openworkflow.sdk:openworkflow-common-grpcand theopenworkflow-commonMaven parent.impl/grpcto common. The common artifact uses grpc-java and Protobuf and has no workflow-engine dependency. Applications select their gRPC transport provider.WorkflowGrpcCallandWorkflowProtobufJsonas public helper names. Reuse nativeChannel, descriptors,MessageandCallOptions; add no request/response DTO hierarchy or conversion framework.Since #1844 is still open, this PR introduces the minimal common parent with the gRPC module only. The shared parent/reactor and root README will need reconciliation when both PRs merge. This PR contains no HTTP implementation changes or HTTP prototype commits.
How it works
WorkflowGrpcCallbuilds native method descriptors and marshallers from resolved Protobuf descriptors. Unary, server-streaming, client-streaming and bidirectional calls returnCompletableFuture<List<Message>>after response completion. Built messages are reused directly; list snapshots copy references, not payloads. JSON conversion throughWorkflowProtobufJsonis explicit.Callers may supply a configured
Channelfor TLS, credentials, interceptors, metadata and connection reuse. That channel is never closed. The address/port convenience overload creates a plaintext channel per invocation and initiates its shutdown when the RPC ends, or forceful shutdown when delivery is abandoned. The library does not block awaiting termination.The original future propagates cancellation to
ClientCall.cancel. LocalorTimeout/completeOnTimeoutand explicit asynchronous completion also stop an unfinished RPC. Cleanup starts before synchronous completion handlers run. Atomic terminal claims preserve the winning outcome during competing cancellation/completion, including the SDK bridge. Supplier-thrown cancellation remains a failed async computation rather than marking the result future cancelled, matching CompletableFuture semantics.CallOptionsare passed unchanged. Standalone defaults remain fail-fast with no deadline; wait-for-ready is opt-in. Native status, cause and trailers are preserved. The SDK keeps its application's executor, wait-for-ready, workflow argument conversion and positioned errors.GrpcExecutor, its public constructor, builder andGRPC_CHANNEL_PROVIDERremain available, with unary and streaming result shapes preserved. Descriptor/resource loading and protoc fallback remain in the implementation module.Validation
After the namespace update (db9a1d5), Spotless, Checkstyle and the full JDK 21 clean Maven build passed locally. Of 832 tests, 815 passed and 17 were skipped. All 44 focused gRPC cases passed.
The original extraction (163750c) was validated locally with Java 21 on macOS ARM64 and Docker:
javac --release 21using only the common gRPC JAR and its non-engine runtime dependencies.The Maven-provided protoc 3.25.8 macOS ARM64 executable was placed on PATH for the existing fallback. No test exclusions, timeout relaxations, failure-ignore or retry settings were added. Intermediate full runs hit timing-sensitive assertions in unchanged scheduler/container tests; those tests pass on untouched main, and the final full clean reactor passes. The 17 skips are existing annotations, including 12 shell cases disabled on macOS.
Trade-offs and limits
Requests are finite buffered lists. Responses accumulate without a size cap and are returned when the stream ends. There is no incremental consumer, application backpressure, durable stream storage or restart/resumption. The SDK still converts collected Protobuf responses after transport completion, which increases peak memory during conversion; its result future releases the upstream reference after adaptation. A per-message mapper is deliberately deferred.
Cancelling a derived future does not automatically cancel its upstream future; coroutine bridges must cancel the original returned future. Cancellation cannot undo server effects already performed. Owned channel shutdown is asynchronous. Prefer caller-managed channels for connection reuse and TLS; TLS negotiation itself is not covered by the plaintext local-server tests.
The existing map-entry convention for client/bidirectional DSL arguments is preserved. Remote Java causes are not serialized by gRPC; locally supplied causes and trailers remain available. The manually triggered integration-tests profile was not run. Local macOS/GraalVM validation does not replace the Ubuntu/Temurin CI result.